1. Field of the Invention
The present invention relates to an image processing apparatus and method and, more particularly, to an image processing apparatus and method which corrects a degraded image using image recovery processing.
2. Description of the Related Art
Since the digitization of information has allowed images to be processed as signal values, there have been proposed various correction processing methods for sensed images. When an object is sensed and imaged by a digital camera, the obtained image is degraded more or less by the aberrations of an optical imaging system, in particular.
The causes of blur components of an image include the spherical aberration, comatic aberration, curvature of field, and astigmatism of an optical system. Each of the blur components of an image due to these aberrations indicates that a light beam emerging from one point of an object is formed into an image with a spread, which should converge into one point on an imaging plane without any aberration or any influence of diffraction. This state is called a PSF (Point Spread Function) in optical terms but will be referred to as a blur component in image terms. A blur of an image may indicate a defocused image but indicates here an image blurred due to the influences of the above aberrations of the optical system even if it is in focus. In addition, color fringing on color images due to the chromatic aberration on the axis, spherical aberration of color, and comatic aberration of color of optical systems can be regarded as different ways of blurring at different wavelengths. Furthermore, color fringing in the lateral direction which are caused by the magnification chromatic aberration of the optical system can be regarded as position shifts or phase shifts due to image sensing magnification differences for the respective wavelengths of light.
The OTF (Optical Transfer Function) obtained by Fourier transform of a PSF is frequency component information of an aberration, which is expressed by a complex number. The absolute value of an OTF, that is, an amplitude component, is called an MTF (Modulation Transfer Function), and a phase component is called a PTF (Phase Transfer Function). That is, an MTF and PTF are respectively the frequency characteristics of an amplitude component and phase component of an image degradation due to aberrations. In this case, a phase component is represented as a phase angle by equation (1):PTF=tan−1(Im(OTF)/Re(OTF))  (1)where Re(OTF) and Im(OTF) respectively represent the real part and imaginary part of the OTF.
As described above, the OTF of an optical imaging system causes degradations in the amplitude component and phase component of an image. For this reason, a degraded image asymmetrically blurs at each point of the object like a comatic aberration.
As a method of correcting degradations in amplitude (MTF) and phase (PTF), a method of correcting them by using the information of the OTF of an optical imaging system is known. This method is called by the terms “image recovery” and “image restoration”. The processing of correcting a degradation in image by using the information of the OTF of an optical imaging system will be referred to as image recovery processing or recovery processing.
The following is an outline of image recovery processing. Letting g(x, y) be a degraded image, f(x, y) be the original image, and h(x, y) be the PSF obtained by inverse Fourier transform of the optical transfer function, equation (2) given below holds:g(x,y)=h(x,y)*f(x,y)  (2)where * represents convolution and (x, y) represents coordinates on the image.
When this equation is converted into a display form on a frequency plane by Fourier transform, it becomes a form of product for each frequency as represented by equation (3):G(u,v)=H(u,v)·F(u,v)  (3)where H is the function obtained by Fourier transform of a PSF, and hence represents an OTF, and (u, v) represents coordinates on a two-dimensional frequency plane, that is, a frequency.
That is, in order to obtain the original image from the sensed degraded image, both sides of equation (3) may be divided by H as represented by equation (4) given below.G(u,v)/H(u,v)=F(u,v)  (4)Returning F(u, v) to a real plane by inverse Fourier transform can obtain the original image f(x, y) as a recovered image.
Letting R be the value obtained by inverse Fourier transform of 1/H in equation (4), it is also possible to obtain the original image by performing convolution processing for an image on the real plane, as represented by equation (5):g(xy)*R(x,y)=f(x,y)  (5)where R(x, y) is called an image recovery filter. An actual image, however, includes noise components. For this reason, using an image recovery filter generated by taking the perfect reciprocal of the OTF in the above manner will amplify noise components together with the degraded image. In general, therefore, a proper image cannot be obtained. In this respect, for example, there is known a method of suppressing the recovery ratio on the high-frequency side of an image in accordance with the intensity ratio between an image signal and a noise signal, such as a method using a Wiener filter. As a method of correcting a degradation in the color fringing component of an image, for example, the degradation is corrected by correcting the above blur components such that the amounts of blurs are made uniform for the respective color components of the image.
In this case, since the OTF varies in accordance with image sensing conditions such as a zooming position and an aperture diameter, and hence it is necessary to change the image recovery filter used for image recovery processing accordingly.
For example, Japanese Patent No. 03532368 discloses a technique of eliminating an image blur in an endoscope for observing the interior of the living body by using a PSF corresponding to a fluorescence wavelength to be used with respect to a range outside the in-focus range of an image sensing apparatus. Since fluorescence is weak, an object optical system with a small f-number is required. This leads to a decrease in focal depth. This technique is therefore designed to obtain an in-focus image by performing image recovery processing with respect to a range in which the optical system goes out of focus.
As described above, performing image recovery processing for a sensed input image can improve image quality by correcting aberrations.
In actual image sensing operation, the sensed state of an input image sometimes does not match the state of an image recovery filter for correcting the sensed state. Consider, for example, that a sensed image has a saturated pixel. The saturated pixel has lost intrinsic object information, and hence the state of the input image does not match that of a degraded image to be processed by an image recovery filter.
When applying a filter for compensating for frequency characteristics to an image as in image recovery processing, the difference in frequency characteristic between the image and the filter may generate ringing in the image. The loss of object information due to saturation makes the saturated region of the sensed image have frequency characteristics greatly different from the intrinsic frequency characteristics of the object. A portion near the boundary between a saturated pixel and unsaturated pixel, in particular, greatly differs in frequency characteristic from the target for the image recovery filter, and hence is a region where ringing tends to occur.
Before a description of ringing generated around a saturated region, an example of an image recovery filter will be described first with reference to FIGS. 15A and 15B schematically showing the image recovery filter. It is possible to determine the number of taps of an image recovery filter in accordance with the aberration characteristics and required recovery accuracy of an optical imaging system. In the case shown in FIG. 15A, a two-dimensional filter with 11×11 taps is used. FIG. 15A omits values (coefficients) in the respective taps. FIG. 15B shows one section of this image recovery filter. The distribution of the values (coefficient values) of the respective taps of the image recovery filter serve to return the PSF, which has spatially spread due to aberrations, to ideally one original point. When using the image recovery filter, the respective taps of the filter are subjected to convolution processing (convolution integration or product sum) in accordance with the respective pixels of the image. In convolution processing, in order to improve the signal value of a given pixel, the pixel is made to coincide with the center of the image recovery filter. This technique then calculates the products of the signal values of the image and the coefficient values of the filter for the respective corresponding pixels of the image and image recovery filter, and replaces the sum total of the products with the signal value of the central pixel.
FIGS. 16A to 16D are graphs showing an example of ringing generated near a saturated portion at the time of the application of an image recovery filter. FIGS. 16A to 16D show pixel values near a given edge in a sensed image. Each abscissa represents the pixel position, and each ordinate represents the pixel value of a given color component. FIG. 16A shows a state before the application of the image recovery filter when there is no saturated pixel. Applying the image recovery filter to this image will generate a recovered image whose edge blur has been corrected, as shown in FIG. 16B. FIG. 16C shows a state before the application of the image recovery filter when saturation has occurred on the high-luminance side of the edge. Even if the image recovery filter is applied to the image in this state, since object information has been lost due to the saturation, the edge blur is not properly corrected. As a result, ringing like that shown in FIG. 16D occurs sometimes.
The conventional technique disclosed in Japanese Patent No. 03532368, however, discloses no technique of reducing ringing generated by pixel saturation.